Beyond Mealy Machines: Learning Translators with Recurrent Neural Networks

Recent work has shown that recurrent neural networks can be trained to behave as nite-state automata from samples of input strings and their corresponding outputs. However, most of the work has focused on training simple networks to behave as the simplest class of deterministic machines, Mealy (or Moore) machines. The class of translations that can be performed by these machines are very limited. For example, input and output strings have the same length. However, deterministic state machines can perform more complex translation tasks, and it has been recently shown that they can be inferred from input{output pairs. In this paper we show how a slightly augmented architecture based on a second-order recurrent neural network may be trained to behave as an instance of the most powerful class of deterministic sequential translator.